Support up-to-date `botocore` and `urllib3 2.0` #1037

jakob-keller · 2023-09-04T12:45:36Z

Description of Change

This PR intends to improve general compatibility of aiobotocore within the Python ecosystem by relaxing the dependency specification of botocore, as well as boto3 and awscli.

Thanks to boto/botocore#3034, this PR also implicitly provides support for urllib3 2.0 for Python 3.10+. Installing urllib3>=2.0.0 causes unit tests to fail due to an outdated development dependency. This PR bumps docker to address the issue.

Assumptions

Upstream diff does not contain any changes that require adjustments to the aiobotocore codebase: Besides JSON data and docstring updates, there are only minor changes to botocore Python code and unit tests, none of which affect aiobotocore. This allows for the proposed widening of the botocore dependency specification.

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

I have read and followed CONTRIBUTING.rst
I have updated test_patches.py where/if appropriate (also check if no changes necessary)
I have ensured that the awscli/boto3 versions match the updated botocore version

codecov · 2023-09-04T12:48:51Z

Codecov Report

Merging #1037 (f62c930) into master (5334722) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1037      +/-   ##
==========================================
- Coverage   86.27%   86.23%   -0.04%     
==========================================
  Files          58       58              
  Lines        5740     5740              
==========================================
- Hits         4952     4950       -2     
- Misses        788      790       +2

Flag	Coverage Δ
unittests	`86.23% <100.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
tests/test_patches.py	`95.91% <ø> (ø)`
tests/test_version.py	`95.28% <100.00%> (-1.89%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

JackKolb-ElationHealth · 2023-10-08T17:13:13Z

I think this will help me with another issue. I'm trying to get a service definition for AWS bedrock supported across both aiobotocore and aioboto3. That requires a botocore bump to atleast 1.31.57.

jakob-keller · 2023-10-08T19:26:14Z

I think this will help me with another issue. I'm trying to get a service definition for AWS bedrock supported across both aiobotocore and aioboto3. That requires a botocore bump to atleast 1.31.57.

That should be possible once this PR is merged and a release is published.

jakob-keller · 2023-10-10T06:06:03Z

Until this PR is reviewed, I intend to monitor botocore releases and push updates to prevent the PR from becoming stale.

just4brown · 2023-10-10T21:14:01Z

Hi team, looks like we have a lot of folks blocked waiting for this change. Any maintainers available to review this? cc @thehesiod @terrycain @jettify

Thanks!

jchacks · 2023-10-12T14:01:30Z

Pinning to a patch version of botocore is very restrictive. Can the upper bound on 'botocore>=1.31.16,<1.31.63', be relaxed to <1.32.0? This is the same as boto3's upper bound on botocore found in https://github.com/boto/boto3/blob/develop/setup.py

jakob-keller · 2023-10-12T14:16:00Z

Pinning to a patch version of botocore is very restrictive.

Agreed.

Can the upper bound on 'botocore>=1.31.16,<1.31.63', be relaxed to <1.32.0?

Unfortunately, this is not possible due to the tight integration between botocore and aiobotocore.

thehesiod · 2023-10-17T15:53:00Z

looking! sorry for delay, have been super busy

thehesiod · 2023-10-17T15:58:02Z

my apologies for delay! kids/work/personal stuff has been honestly overwhelming this last month. Deploying shortly

jakob-keller · 2023-10-17T15:58:48Z

my apologies for delay! kids/work/personal stuff has been honestly overwhelming this last month. Deploying shortly

We hear you! 😄

thehesiod · 2023-10-17T16:05:23Z

FYI merging 2.7.0 + 2.7.1 in #1034 as 2.7.0 was never released

jakob-keller · 2023-10-17T16:26:40Z

FYI merging 2.7.0 + 2.7.1 in #1034 as 2.7.0 was never released

Sorry, my bad!

paulmueller · 2023-10-17T20:17:52Z

Sorry for posting on a closed PR. Is there any reason why botocore was pinned to

botocore<1.31.65,>=1.31.16

instead of

botocore<=1.31.65,>=1.31.16

The problem I am having is that boto3 1.28.65 requires botocore<1.32.0,>=1.31.65. I assume there is a breaking change in botocore 1.31.65?

So I am stuck with urllib3<2 and cannot use the latest versions of aiobotocore and s3fs.

jakob-keller · 2023-10-17T20:26:09Z

Sorry for posting on a closed PR. Is there any reason why botocore was pinned to

botocore<1.31.65,>=1.31.16

instead of

botocore<=1.31.65,>=1.31.16

botocore==1.31.65 was released after this PR was prepared and merged. Supporting botocore==1.31.65 would be trivial, but requires a separate PR and release.

You should be able to gain the benefits of this PR, if you work with boto3<=1.28.64.

paulmueller · 2023-10-17T21:07:17Z

Yes thanks, you are correct. The install works, albeit with a quite old version of s3fs (so I am waiting for s3fs to support aiobotocore 2.7.0).

$ pip install boto3==1.28.64 aiobotocore s3fs "urllib3>=2.0"
$ pip freeze
aiobotocore==2.7.0
aiohttp==3.8.6
aioitertools==0.11.0
aiosignal==1.3.1
async-timeout==4.0.3
attrs==23.1.0
boto3==1.28.64
botocore==1.31.64
charset-normalizer==3.3.0
frozenlist==1.4.0
fsspec==2021.7.0
idna==3.4
jmespath==1.0.1
multidict==6.0.4
python-dateutil==2.8.2
s3fs==2021.7.0
s3transfer==0.7.0
six==1.16.0
urllib3==2.0.7
wrapt==1.15.0
yarl==1.9.2

potiuk · 2023-10-19T19:21:40Z

Yes thanks, you are correct. The install works, albeit with a quite old version of s3fs (so I am waiting for s3fs to support aiobotocore 2.7.0).

Same here. Already merged in fsspec/s3fs#809

potiuk · 2023-10-19T19:24:28Z

And.... THANKS. This one will solve a lof of our problems in Apache Airflow ! We had to figure out a very weird scheme there to cope with pinning.

jakob-keller force-pushed the relax-botocore-pin branch 2 times, most recently from adce0a0 to a0496c2 Compare September 10, 2023 19:41

jakob-keller force-pushed the relax-botocore-pin branch 9 times, most recently from 383ba1d to eebf721 Compare September 20, 2023 21:33

jakob-keller force-pushed the relax-botocore-pin branch 3 times, most recently from ae4143c to ea5d1fc Compare October 2, 2023 01:09

andrii-i mentioned this pull request Oct 3, 2023

update botocore to latest version; update awscli, boto3 accordingly #1042

Closed

5 tasks

jakob-keller force-pushed the relax-botocore-pin branch 3 times, most recently from d9117cd to 7459d16 Compare October 6, 2023 04:08

JackKolb-ElationHealth mentioned this pull request Oct 8, 2023

Feature Request: New Service support for "bedrock-runtime" terricain/aioboto3#317

Closed

mraspaud mentioned this pull request Oct 9, 2023

Update versioneer pytroll/pytroll-collectors#144

Merged

jakob-keller force-pushed the relax-botocore-pin branch 2 times, most recently from 6904c30 to a7b2f7c Compare October 9, 2023 15:00

relax botocore dependency specification

2aba86a

jakob-keller force-pushed the relax-botocore-pin branch from a7b2f7c to c30e596 Compare October 9, 2023 15:23

bump minimum docker-py

724dd28

jakob-keller force-pushed the relax-botocore-pin branch from c30e596 to 724dd28 Compare October 9, 2023 15:33

jakob-keller changed the title ~~relax botocore dependency specification~~ Support up-to-date botocore and urllib3 2.0 Oct 10, 2023

relax botocore dependency specification

f62c930

jakob-keller force-pushed the relax-botocore-pin branch from 4e0ce67 to f62c930 Compare October 17, 2023 06:56

thehesiod self-requested a review October 17, 2023 15:54

thehesiod approved these changes Oct 17, 2023

View reviewed changes

thehesiod merged commit 9dd517b into aio-libs:master Oct 17, 2023
11 checks passed

jakob-keller deleted the relax-botocore-pin branch October 17, 2023 15:57

paulmueller mentioned this pull request Oct 17, 2023

bump aiobotocore to 2.6.0 fsspec/s3fs#801

Closed

lou-k mentioned this pull request Oct 18, 2023

Update requirements.txt For aiobotocore 2.7 fsspec/s3fs#809

Merged

bolkedebruin mentioned this pull request Oct 19, 2023

AIP-58: Add Airflow ObjectStore (AFS) apache/airflow#34729

Merged

phillipuniverse mentioned this pull request Oct 25, 2023

Bump aiobotocore from 2.6.0 to 2.7.0 terricain/aioboto3#319

Closed

aws-tianquaw mentioned this pull request Sep 25, 2024

aiobotocore dependency on old version of botocore causes dependency conflicts aws/sagemaker-distribution#83

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support up-to-date `botocore` and `urllib3 2.0` #1037

Support up-to-date `botocore` and `urllib3 2.0` #1037

jakob-keller commented Sep 4, 2023 •

edited

Loading

codecov bot commented Sep 4, 2023 •

edited

Loading

JackKolb-ElationHealth commented Oct 8, 2023 •

edited

Loading

jakob-keller commented Oct 8, 2023

jakob-keller commented Oct 10, 2023

just4brown commented Oct 10, 2023

jchacks commented Oct 12, 2023 •

edited

Loading

jakob-keller commented Oct 12, 2023

thehesiod commented Oct 17, 2023

thehesiod commented Oct 17, 2023

jakob-keller commented Oct 17, 2023

thehesiod commented Oct 17, 2023

jakob-keller commented Oct 17, 2023

paulmueller commented Oct 17, 2023 •

edited

Loading

jakob-keller commented Oct 17, 2023

paulmueller commented Oct 17, 2023 •

edited

Loading

potiuk commented Oct 19, 2023

potiuk commented Oct 19, 2023 •

edited

Loading

Support up-to-date botocore and urllib3 2.0 #1037

Support up-to-date botocore and urllib3 2.0 #1037

Conversation

jakob-keller commented Sep 4, 2023 • edited Loading

Description of Change

Assumptions

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

codecov bot commented Sep 4, 2023 • edited Loading

Codecov Report

JackKolb-ElationHealth commented Oct 8, 2023 • edited Loading

jakob-keller commented Oct 8, 2023

jakob-keller commented Oct 10, 2023

just4brown commented Oct 10, 2023

jchacks commented Oct 12, 2023 • edited Loading

jakob-keller commented Oct 12, 2023

thehesiod commented Oct 17, 2023

thehesiod commented Oct 17, 2023

jakob-keller commented Oct 17, 2023

thehesiod commented Oct 17, 2023

jakob-keller commented Oct 17, 2023

paulmueller commented Oct 17, 2023 • edited Loading

jakob-keller commented Oct 17, 2023

paulmueller commented Oct 17, 2023 • edited Loading

potiuk commented Oct 19, 2023

potiuk commented Oct 19, 2023 • edited Loading

Support up-to-date `botocore` and `urllib3 2.0` #1037

Support up-to-date `botocore` and `urllib3 2.0` #1037

jakob-keller commented Sep 4, 2023 •

edited

Loading

codecov bot commented Sep 4, 2023 •

edited

Loading

JackKolb-ElationHealth commented Oct 8, 2023 •

edited

Loading

jchacks commented Oct 12, 2023 •

edited

Loading

paulmueller commented Oct 17, 2023 •

edited

Loading

paulmueller commented Oct 17, 2023 •

edited

Loading

potiuk commented Oct 19, 2023 •

edited

Loading